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1 Introduction 

One of the most recalcitrant unsolved problems in operator theory is the invariant subspace problem. The 
question has an easy formulation. Does every operator on an infinite dimensional, separable complex Hilbert 
space have a non trivial invariant subspace? 

Despite the simplicity of its statement, this is a very difficult problem and it has generated a very large 
amount of literature. We refer the reader to the expository paper of Yadav [TJ for a detailed account of 
results related to the invariant subspace problem. 

In this survey we discuss some applications of fixed point theorems in the theory of invariant subspaces. 
The general idea is that a fixed point theorem applied to a suitable map yields the existence of invariant 
subspaces for an operator on a Banach space. 
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In Section[5]we consider the striking theorem of Lomonosov [2 about the existence of invariant subspaces 
for algebras containing compact operators. The proof of this theorem is based on the Schauder hxed point 
theorem. 

In Section |3] we present a recent result of Lomonosov, Radjavi, and Troitsky [3] about the existence 
of invariant subspaces for localizing algebras. The proof of this result is based on the Ky Fan fixed point 
theorem for multivalued maps. The idea of using hxed point theorems for multivalued maps in the search 
for invariant subspaces was first introduced by Androulakis pQ. 

In Section[4]we consider an extension of Burnside's theorem to infinite dimensional Banach spaces. This 
result is originally due to Lomonosov [5]. We present a proof of it in a special case that was obtained 
independently by Scott Brown [6] and that once again is based on the Schauder fixed point theorem. 

In Section [5] we address the existence of invariant subspaces for operators on Krein space of indefinite 
product, and we present a result of Albeverio, Makarov, and Motovilov [7] whose proof uses the Banach 
fixed point theorem. 

The rest of this section contains some notation, a precise statement of the invariant subspace problem, 
and a few historical remarks. 

Let E be an infinite dimensional, complex Banach space and let B(E) denote the algebra of all bounded 
linear operators on E. A subspace of E is by definition a closed linear manifold in E. 

A subspace M C E is said to be invariant under an operator T £ B(E) provided that TM C M, and a 
subspace M C E is said to be invariant under a subalgebra TZ C B(H) provided that M is invariant under 
every R £ TZ. A subalgebra TZ C B(H) is said to be transitive provided that the only subspaces invariant 
under TZ are the trivial ones, M = {0} and M = E. This is equivalent to say that the subspace {Rx : R £ TZ} 
is dense in E for each x £ E\{0}. 

The commutant of a set of operators S C B{E) is the subalgebra S' of all operators R £ B(E) such 
that SR = RS for all S £ S. A subspace M C E is said to be hyperinvariant under an operator T £ B(E) 
provided that M is invariant under {T}'. 

The invariant subspace problem is the question of whether every operator in B(E) has a non trivial 
invariant subspace. This is one of the most important open problems in operator theory. 

The origin of this question goes back to 1935, when von Neumann proved the unpublished result that any 
compact operator on a Hilbert space has a non trivial invariant subspace. Aronszajn and Smith [5] extended 
this result in 1954 to general Banach spaces. Bernstein and Robinson [3] used non standard analysis to prove 
in 1966 that every polynomially compact operator on a Hilbert space has a non trivial invariant subspace. 
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Halmos [TU] obtained a proof of the same result using classical methods. 

Lomonosov [2] proved in 1973 that any non scalar operator on a Banach space that commutes with a 
non zero compact operator has a non trivial hyperinvariant subspace. The result of Lomonosov came into 
the scene like a lightning bolt in a clear sky, generalizing all the previously known results, and introducing 
the use of the Schauder fixed point theorem as a new technique to produce invariant subspaces. 

Enflo jllj constructed in 1976 the first example of an operator on a Banach space without non trivial 
invariant subspaces. The example circulated in preprint form and it did not appear published until 1987, 
when it was recognized as correct work [12] , In the meantime, Beauzamy |13j simplified the technique, and 
further examples were given by Read |141[T5] . 

Very recently, Argyros and Haydon [16j constructed an example of an infinite dimensional, separable 
Banach space such that every continuous operator is the sum of a compact operator and a scalar operator, 
so that every operator on it has a non trivial invariant subspace. 

However, after so many decades, the question about the existence of invariant subspaces for operators 
on Hilbert space is still an open problem. 

2 Invariant subspaces for algebras containing compact operators 

We start with a fixed point theorem that is the key for the main result in this section. The use of this result 
is one of the main ideas in the technique of Lomonosov. We shall denote by conv(S') the closed convex hull 
of a subset S C E. 

Proposition 2.1. Proposition 1] Let E be a Banach space, let C C E be a closed convex set, and let 
$ : C — > E be a continuous mapping such that $((7) is a relatively compact subset of C. Then there is a 
point xq £ C such that <&(xq) — xq. 

Proof. Let Q denote the closure of $(C). It follows from a theorem of Mazur that conv((5) is a compact, 
convex subset of E, and since C is closed and convex, we have conv(Q) C C. Since $(C) C Q, we have 
<&(conv(Q)) C Q C conv(Q), and now the result follows from the Schauder fixed point theorem. □ 

Theorem 2.2. |77j Theorem 2] Let K C B(E) be a transitive algebra and let K e 13(E) be a non zero 
compact operator. Then there is a operator R € 1Z and there is a vector x\ € E such that RKx\ = x\. 

Proof. We may assume without loss of generality that = 1. Choose an xo £ E such that H-fCxoH > 1> 
so that ||xo|| > 1. Consider the closed ball B = {x E E : \\x — xq\\ < 1}. Then, for each R 6 1Z, consider the 
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open set Gr — {y £ E : \\Ry — soil < 1}- Since 72. is a transitive algebra, we have 

|J G R = E\{0}. 

Ren 

Since K is a compact operator, KB is a compact subset of E, and since ||i^|| = 1 and ||-K"xo|| > 1, we have 
^ KB. Thus, the family {Gr : R € 11} is an open cover of KB. Hence, there exist finitely many operators 
Ri , . . . , R n £ 1Z such that 

n 

Be [jG Ri . 

i=l 

Next, for each y £ KB and i = 1, . . . , n we define on(y) = max{0, 1 — \\Riy — £o||}- Then < on(y) < 1, and 

n 

for each y £ KB there is an i = 1, . . . , n such that y £ G Ri , so that «i(y) > 0. Thus ai(y) > for each 

i=i 

y £ KB, and we may define 



AGO 



for i = 1, . . . , n and y £ KB. Now, each Pi is a continuous function from KB into M. Hence, we may define 
a continuous mapping <f> : _B — > E by the expression 

n 

$( x ) = ^fi l {Kx)R l Kx. 

i=l 

n 

We claim that C _B. Indeed, for each x £ B we have ^/3; (if x) = 1, so that 



i=i 



\<f>(x)-x \\ 



Y / ^(Kx)(R l Kx-x ) 



<J2MKx)\\R t Kx-x a \ 



If \\RiKx — xq\\ > 1 then cti(Kx) = and therefore f3i(Kx) = 0. Hence 

n 

\Mx)-x \\<^2/3 i (Kx) = l > 
and this completes the proof of our claim. Finally, each operator R{K is compact, so that each RiKB is 

n 

relatively compact, and it follows from an earlier mentioned theorem of Mazur that Q = conv RiKB is 

i=i 

compact. Since $(£>) C Q, the set is a relatively compact subset of B. Now we apply Proposition [^J] 

to find a vector x\ £ B such that $(xi) = £1. Since ^ B, we have xi ^ 0. Then we consider the operator 
defined by 

n 

Rx = ^Tp i (Kx 1 )R i x, 



and we conclude that R £lZ and RKx\ = x\, as we wanted. 



□ 
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Corollary 2.3. f^, Theorem 3} Every non scalar operator that commutes with a non zero compact 
operator has a non trivial, hyperinvariant subspace. 

Proof. Let T £ 13(E) be a non scalar operator and suppose that T commutes with a nonzero compact 
operator K. We must show that the commutant {T}' is non transitive. Suppose, on the contrary, that {T}' 
is transitive. We can apply Theorem l2.2l to hnd an operator R £ {T}' such that A = 1 is an eigenvalue of the 
compact operator RK with associated finite dimensional eigenspace F — kei (RK — I). Since T commutes 
with RK, we observe that T maps F into itself and therefore T must have an eigenvalue. Since T is non scalar, 
the corresponding eigenspace M cannot be the whole E, and it is invariant under {T}'. The contradiction 
has arrived. □ 

3 Invariant subspaces for localizing algebras 

In this section we use the following fixed point theorem of Ky Fan [18]. Recall that if O is a topological 
space and $ : fl — ► V(£l) is a point to set map from f2 to the power set of fi, then $ is said to be upper 
semicontinuous if for every xo £ f2 and every open set U C Q such that <f>(xo) C U there is a neighborhood 
V of xq such that <f>(x) C U for every x G V. In terms of convergence of nets, this definition is equivalent to 
say that for every x £ Q, for every net (x a ) with x a — > x, and for every y a £ <5>(x a ) such that the net (y a ) 
converges to some y £ fl, we have y £ 

Theorem 3.1 (Ky Fan fixed point theorem |18j). Let C be a compact convex subset of a locally convex space 
and let $ : C — > V(C) be an upper semicontinuous mapping such that is a non empty, closed convex 

set for every x £ C. Then there is an Xq £ C such that Xq £ $(xo). 

A subalgebra 1Z C B(H) is said to be strongly compact if its unit ball is precompact in the strong operator 
topology. An important example of a strongly compact algebra is the commutant of a compact operator 
with dense range. We shall denote by ball(7?.) the unit ball of 7Z. 

This notion was introduced by Lomonosov [19] as a means to prove the existence of invariant subspaces 
for essentially normal operators on Hilbert spaces. Recall that an operator T on a Hilbert space is said to 
be essentially normal if T*T — XT* is a compact operator. Lomonosov showed that if an essentially normal 
operator T has the property that both its commutant {T}' and the commutant of its adjoint {T*}' fail to 
be strongly compact, then T has a nontrivial invariant subspace. 

Thus, in order to solve the invariant subspace problem for essentially normal operators, it suffices to 
consider only operators with a strongly compact commutant. 
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Lomonosov, Radjavi and Troitsky [3] obtained a result about the existence of invariant subspaces for an 
operator with a strongly compact commutant under the additional assumption that the commutant of the 
adjoint is a localizing algebra. 

A subalgebra TZ C B(E) is said to be localizing provided that there is a closed ball B C E such that ^ B 
and such that for every sequence (x n ) in B there is a subsequence (x nj ) and a sequence of operators (Rj) in 
TZ such that \\Rj\\ < 1 and (RjX nj ) converges in norm to some non zero vector. An important example of a 
localizing algebra is any algebra containing a non zero compact operator. 

Proposition 3.2. J3J Proof of Theorem 2.3] Let TZ C B(E) be a transitive localizing algebra, let B C E be 
a closed ball as above, and let T G TZ' be a nonzero operator. Then there exists an r > such that for every 
x G B we have r ball (ft) (IV) flB^U. 

Proof. First, T is one to one, because 1Z is transitive and kerT is invariant under TZ. If this is not so, then 
for every n > 1 there is a vector x n € B such that ||i?|| > n, whenever R e TZ and RTx n G B. Since TZ is 
localizing, there is a subsequence {x nj ) and a sequence (Rj) in TZ such that ||-Rj|| < 1 and (RjX rij ) converges 
in norm to some nonzero vector x G X. We have TRj — RjT for all j > 1, so that (RjTx nj ) converges 
to Tx in norm. Now Tx ^ because T is injective and x ^ 0. Since 7?. is transitive, there is an operator 
R G TZ such that i?T;r G int £>. It follows that there is a jo > 1 such that RRfTx nj G int _B for every j > jo. 
Since RRj € 7?., the choice of the sequence (:r ra ) implies that ||_Ri?j|| > nj for every j > jo, and this is a 
contradiction because < \\R\\ for every j >1. □ 

If is a Banach space then E* denotes its dual space. If TZ C B(E) is a subalgebra, then 7?.* denotes 
the subalgebra of B(E*) of the adjoints of the elements of TZ, that is TZ* = {R* : R G TZ}. 

Theorem 3.3. Theorem 2.3] Let E be a complex Banach space, let TZ G B(E) be a strongly compact 
subalgebra such that TZ* is a transitive localizing algebra and it is closed in the weak-* operator topology. If 
T G TZ' is a non zero operator then there is an operator R G TZ and there is a non zero vector x* G E* such 
that R*T*x* — x* . Moreover, the operator T* has a non trivial invariant subspace. 

Proof. We shall apply Proposition 13.21 to the algebra TZ* . Let B* C E* be a closed ball as in the definition 
of a localizing algebra, let r > be a positive number as in Proposition 13.21 and define a multivalued map 
$ : B* — > V{B*) by the expression 

= rbal\{TZ*)(T*x*) n B* . 



G 



Then, $(x*) is a non empty, convex subset of B*. Also, $(x*) is weak-* closed because baH(1Z*)(T*x*) is 
weak-* compact as the image of ball(7£*) under the map R* — > R*T*x*, which is continuous from B(E*) 
with the weak-* operator topology into E* with the weak-* topology, and ball(7?.*) is compact in the weak-* 
operator topology. 

We claim that <E> is upper semicontinuous for the weak-* topology. Indeed, let x*,y* G B* , and let (x*) 
and (y*) be two nets in B* with x* a ->• x* , y* -> y* in the weak-* topology, and such that y* € We 
must show that y* G $(x*). Since y* <E $(a£), there is an i?* G ball(7?.*) such that y* = rR* a T*x* a . Since 
ball(7?.) is precompact in the strong operator topology, there exists a subnet (i? Q/3 ) that converges in the 
strong operator topology to some R G B(E). Thus, R* afj — > R* in the weak-* operator topology. Notice that 
baH(TV) is compact in this topology because b&\\(B(E*)) is compact in this topology and TV is closed in 
this topology. It follows that R* G ball(7£*). Let x G E and notice that \\TR a? x - TRx\\ -> 0. Then 

(x,y^) = (x,ri?^T*^) 
= r(TR a pX, x a p) 

= r(TR a/3 x -TRx,x* ap ) +r(TRx,x* afi }. 
We have (TR aft x - TRx,x* ap ) -> and (TRx,x* afi ) ~> (TRx,x*) = (x,R*T*x*), so that 

(as.y^) -> (x,rR*T*x*). 

Since x G £ is arbitrary, y*^ — > rR*T*x* in the weak-* topology, and it follows that y* = rR*T*x*. This 
shows that y* G $(x*), and the proof of our claim is complete. 

Since the map $ is upper semicontinuous and B* is compact in the weak-* topology, it follows from the 
Ky Fan fixed point theorem that there is a vector x* G B* such that x* G $(x*), that is, there is an operator 
R G ball(ft) such that x* = rR*T*x*. □ 

Corollary 3.4. Corollary 2.4] Let T G B(E) be an operator such that {T}' is a strongly compact algebra 
and {T*}' is a localizing algebra. Then T* has a nontrivial invariant subspace. 

Proof. If T* has a hyperinvariant subspace then there is nothing to prove, and otherwise {T*}' is a transitive 
algebra, so that Theorem I3.3I applies. □ 

Notice that the assumptions of Corollary [33] are met whenever T is a compact operator with dense range. 
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4 An infinite dimensional version of Burnside's theorem 

Burnside's classical theorem is the assertion that for a finite dimensional linear space F, the only transitive 
subalgebra of B(F) is the whole algebra B(F). Lomonosov [5] obtained a generalization of Burnside's theorem 
to infinite dimensional Banach spaces. Scott Brown [5] proved the same result independently for the special 
case of a Hilbert space and a commutative algebra. Lindstrom and Schluchtermann |20j provided a relatively 
short proof of the Lomonosov result in full generality. In this section we present a proof of the Scott Brown 
result that is based on Schauder fixed point theorem. 

Let H be a complex, infinite dimensional, separable Hilbert space. Let T € B(H) and let ||T|| e denote 
the essential norm of T, that is, the distance from T to the space of compact operators. 

Theorem 4.1. J51 Theorem 1.1] Let 1Z be a commutative subalgebra of B(H). Then there exist nonzero 
vectors x,y £ H such that for any R £ 1Z we have \(Rx,y}\ < ||i?|| e - 

Proof. Consider the set £ = {R € 1Z : \\R\\ e < 1/16}. We claim that there is some x £ H\{0} such that 
the set Ex is not dense in H. The result then follows easily because in that case there is some y £ H\{0} 
such that | (Rx, y) \ < 1 for all R £ £. Now, for the proof of our claim, we proceed by contradiction. Suppose 
that the set £x is dense in H for every x £ H\{0}. Choose xo £ H with ||xo|| = 2 and consider the closed 
ball B = {x £ H : \\x — xq\\ < 1}. Then, for every vector x £ B there is an operator R x £ £ such that 
H-R^a; — io|| < 1/2- Next, there is a bounded operator T x and a compact operator K x such that R x = T x + K x 
and \\T X \\ < 1/8. Since K x is a compact operator, it is weak to norm continuous on bounded sets, so that 
there exists an open neighborhood of x in the weak topology, say V x C H, such that \\K x y — K x x\\ < 1/4 
for all y £ V x OB. Then consider the set U x = V x fl B and notice that U x is an open neighborhood of x in 
the weak topology relative to B. Moreover, for y £ U x we have 



and therefore \\R x y — xq\\ < 1. Hence, R X U X C B. Since B is compact in the weak topology, there exist 
finitely many vectors x\, . . . x n £ B such that 



Choose some weakly continuous functions /i, . . . , f n on B such that supp(/j) C Uj, < fj(x) < 1, and 



\\RxV ~ R*x\\ < \\T x y - T x x\\ + \\K x y - K x x\\ < 2 • - + - = -, 



n 



BQ\Ju Xj . 



n 




for all x £ B. 



3=1 



S 



Define a weakly continuous mapping $ : B — > £> by the expression 

n 

$(#) = fj(x)R Xj x for all x £ B, 

and apply the Schauder fixed point theorem to find a vector y Q £ B such that $(j/o) =2/0- Finally consider 
the operator R £lZ defined by the expression 

n 
3 = 1 

Hence, i?y = J/o- Notice that R ^ I because ||-R|| e < 1/8. Then, the eigenspace M = {x £ H : Rx = x} is 
a closed nontrivial invariant subspace for the algebra 1Z. Thus, any vector x £ M has the property that the 
set Ex is not dense in H. The contradiction has arrived. □ 

5 Invariant subspaces for operators on Krein space 

Let Hi, H2 be two Hilbert spaces and consider the orthogonal direct sum H = Hi © H2. Let Pi, P2 denote 
the orthogonal projections from H onto Hi, Hi, respectively. Consider the operator J := Pi — Pi- The Krein 
space is the space H provided with the indefinite product 

[x,y] := {Jx,y), x,y e H. 

Notice that J is a selfadjoint involution, that is, J* = J and J 2 = /. The operator J is sometimes called the 
fundamental symmetry of the Krein space. 

A vector x e H is said to be non negative provided that [x, x] > 0, and a subspace M C H is said to be 
non negative provided that [x, x] > for all x € M. 

Every operator T g has a matrix representation 

y _ Tn T12 
T21 T22 

with respect to the decomposition H = Hi © H 2 - 

There is a natural, one to one and onto correspondence between the maximal non negative invariant 
subspaces M of an operator T e B(H) and the contractive solutions X £ B{Hi,H2) of the so called operator 
Riccati equation 

XT12X + XTu - T22X - T 2 i = 0. 

The correspondence XnMis given by M — {xi Xxi : Xi £ Hi}, where \\X\\ < 1. The operator T is 
usually called the hamiltonian operator of the operator Ricatti equation. 
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An operator T s B(H) is said to be J- self adjoint provided that [Tx, y] — [x. Ty] for every x,y € H. This 
is equivalent to say that JT = T*J, or in other words, Tj^ = Tn, T 22 = T 22 , and T* 2 — —T 2 x- 

A classical theorem of Krein is the assertion that, if the hamiltonian operator T is J-selfadjoint and the 
corner operator T\ 2 is compact, then there exists a maximal non negative invariant subspace for T. 

Albeverio, Makarov, and Motovilov [7] addressed the question of the existence and uniqueness of con- 
tractive solutions to the operator Riccati equation under the condition that the diagonal entries in the 
hamiltonian operator have disjoint spectra, that is, ct(Th) (1 cr(T 22 ) — 0- They proved the following 

Theorem 5.1. ^ Theorem 3.6 and Lemma 3.11] There is some universal constant c > such that whenever 
the corner operator T\ 2 satisfies the condition 

\\T 12 \\ <c-dist[a(T n ),a(T 22 )}, 

there is a unique solution X to the operator Riccati equation with \\X\\ < 1. 

An earlier result in this direction was given by Motovilov [2T| Corollary 1] with the stronger assumption that 
the corner operator T\ 2 is Hilbert-Schmidt. Adamjan, Langer, and Tretter [22] extended the technique to 
the case that the hamiltonian operator is not J-selfadjoint. Kostrykin, Makarov, and Motovilov [23] adopted 
the assumption that ct(Th) lies in a gap of o~(T 22 ) and they showed that the best constant, in that context, 
is c = y/2. 

We present a proof of Theorem 15.21 that is based on Banach fixed point theorem. This method can be 
found in the paper of Albeverio, Motovilov and Shkalikov [53J Theorem 4.1]. A basic tool is the bounded 
linear operator R defined for X € B(Hi, H 2 ) by the expression 

R(X) :=T 22 X-XT n . 

It follows from the Rosenblum theorem that the map R is invertible. The main result is the following 

Theorem 5.2. \24\ Theorem 4-1] If the operators Tn,T 22 have disjoint spectra and the corner operator T\ 2 
satisfies the estimate 

l|T 12 || < ^ 

then there is a unique solution X to the operator Riccati equation with \\X\\ < 1. 

The following upper bound on the norm of the inverse can be found in the work of Albeverio, Makarov, 
and Motovilov {7, Theorem 2.7]. See also the paper by Bhatia and Rosenthal [4, p. 15] for this interesting 
result and other related issues. 
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Theorem 5.3. Theorem 2.7] If the operators Tn,T 22 have disjoint spectra then 

MD-in < * 1 

Notice that Theorem 15.11 becomes a corollary of Theorem 15.21 and Theorem 15.31 with the constant c = 1/tt. 
Proof of Theorem \5.2[ Consider the quadratic map Q defined for X E B(Hi,H 2 ) by the expression 

Q{X) :=XT 12 X -T 21 . 
It is clear that the operator Riccati equation can be expressed as 

Q(X) - R(X) = 0, 

or equivalently, X — R^ 1 (Q(X)). Thus, the solutions of the operator Riccati equation are the fixed points 
of the map S := Br 1 o Q. Now, let us check that the map S takes the unit ball of B(Hi,H2) into itself. 
Indeed, if ||X|| < 1 then 

= WR-'iQiX))]] < WR- 1 ]] ■ \\Q(X)\\ 

< II^H-diriall-lW + ll^ill) 

< ||i?- 1 ||-(||r 12 || + ||T 21 ||) = 2p- 1 ||-||r 12 ||<i. 

Also, the map S is contractive, for if \\X\\, \\Y\\ < 1 then 

\\Q(X) - Q(Y)\\ = \\XT 12 X -YT 12 Y\\ 

< \\XT 12 X - XT 12 Y\\ + \\XT 12 Y - YT 12 Y\\ 

< (||x|| + ||r||).||T 12 ||.||x-r||<2||T 12 ||.||x-r||, 

and from this inequality it follows that 

\\S(X)-S(Y)\\ = \\R-i(Q(X)-Q(Y))\\ 

< wr-'w ■ \\Q(x) - Q0OII < ^l^" 1 !! • ll T i2ll -11^-^11, 

so that the map S satisfies a Lipschitz condition with a Lipschitz constant 2||i?~ 1 || ■ ||T 12 || < 1. The result 
now follows at once as a consequence of the Banach fixed point theorem. □ 
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